Papers with RNN LMs
Recurrent Neural Network Language Models Always Learn English-Like Relative Clause Attachment (2020.acl-main)
Copied to clipboard
| Challenge: | Language modeling is widely used as pretraining for many tasks involving language processing. |
| Approach: | They extend a standard approach to evaluating language models to cases of valid interpretations . they compare model performance in English and Spanish to show that non-linguistic bias overlaps with syntactic structure in English but not Spanish. |
| Outcome: | The proposed model performance in English and Spanish shows that non-linguistic bias overlaps with syntactic structure in English but not in Spanish. |
On Efficiently Representing Regular Languages as RNNs (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work by Hewitt et al. (2020) provides an interpretation of the empirical success of recurrent neural networks (RNNs) as language models (LMs). |
| Approach: | They generalize their construction and show that RNNs can efficiently represent a larger class of LMs than previously claimed. |
| Outcome: | The results suggest that RNNs can represent a larger class of LMs than previously claimed . |
Lower Bounds on the Expressivity of Recurrent Neural Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies of the representational capacity of neural LMs have focused on their ability to recognize formal languages. |
| Approach: | They propose to connect recurrent neural networks (RNNs) as classifiers to finite-state automatas (FSAs) and a probabilistic FSA to characterize their representational capacity. |
| Outcome: | The proposed models can express arbitrary regular LMs with linearly bounded precision. |
On the Representational Capacity of Recurrent Neural Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on LMs as formal languages, but they do not consider language membership. |
| Approach: | They extend the Turing completeness result to the probabilistic case . they show that a rationally weighted RLM can simulate any deterministic Turing machine . |
| Outcome: | The proposed model can simulate any deterministic Turing machine with rationally weighted transitions . the proposed model is based on recurrent neural networks with a rational weighting over strings . |